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CLAIMS 

What is claimed is: 

1. A computer-implemented method of reducing graphical user interface 
(GUI) noise comprising: 

recording a first execution scenario for control of operation of an application 
program having a GUI during a recording phase of operation of a cognitive control 
framework system; 

setting soft conditions for a search for the application program for the first 
execution scenario; 

playing back the application program according to the first execution scenario 
during a playback phase of operation of the cognitive control framework system; 

updating the first execution scenario to form a second execution scenario to 
reduce GUI noise conditions observed during playback, including updating recorded 
images originally generated by the GUI during the recording phase and updating 
coordinates for user input data; 

setting stronger conditions for the search for use in subsequent playbacks; and 

playing back the application program according to the second execution scenario 
with the stronger conditions for search. 

2. The method of claim 1, wherein the soft conditions comprise a first set of 
bounds for differences in shapes of contours, text and image content, or layout. 

3. The method of claim 2, where the stronger conditions comprise a second set 
of bounds for differences in shapes of contours, text and image content, or layout, the 
second set being different than the first set. 

4. The method of claim 1, wherein the user input data comprises mouse 
selections. 



5. The method of claim 1, wherein GUI noise conditions comprise at least one 
changeable color schemes, highlighting of items, noise from video sources, and anti- 



24 



aliasing effects. 

6. The method of claim 1, wherein updating recorded images comprises using 
playback images as recorded images for subsequent playbacks. 

7. An article comprising: a machine accessible medium containing instructions, 
which when executed, result in reducing graphical user interface (GUI) noise by 

recording a first execution scenario for control of operation of an application 
program having a GUI during a recording phase of operation of a cognitive control 
framework system; 

setting soft conditions for a search for the application program for the first 
execution scenario; 

playing back the application program according to the first execution scenario 
during a playback phase of operation of the cognitive control framework system; 

updating the first execution scenario to form a second execution scenario to 
reduce GUI noise conditions observed during playback, including updating recorded 
images originally generated by the GUI during the recording phase and updating 
coordinates for user input data; 

setting stronger conditions for the search for use in subsequent playbacks; and 
playing back the application program according to the second execution scenario 
with the stronger conditions for search. 

8. The article of claim 7, wherein the soft conditions comprise a first set of 
bounds for differences in shapes of contours, text and image content, or layout. 

9. The article of claim 8, where the stronger conditions comprise second set of 
bounds for differences in shapes of contours, text and image content, or layout, the 
second set being different than the first set. 



10. The article of claim 7, wherein the user input data comprises mouse 
selections. 
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1 1 . The article of claim 7, wherein GUI noise conditions comprise at least one 
of changeable color schemes, highlighting of items, noise from video sources, and anti- 
aliasing effects. 

12. The article of claim 7, wherein instructions to update recorded images 
comprise instructions to use playback images as recorded images for subsequent 
playbacks. 

13. A method of automatically controlling execution of an application program 
having a GUI to reduce GUI noise comprising: 

capturing user input data and images displayed by the GUI during a recording 
phase of execution of the application program; 

analyzing the captured user input data and recorded images to generate a first 
execution scenario during the recording phase; 

setting soft conditions for a search for the application program for the first 
execution scenario; 

generating simulated user input data based on the first execution scenario during 
a playback phase of execution of the application program and inputting the simulated 
user input data to the application program; 

performing image analysis on playback images displayed by the GUI as a result 
of processing the simulated user input data during the playback phase and the recorded 
images; 

updating the first execution scenario to form a second execution scenario to 
reduce GUI noise conditions observed during playback, including updating the recorded 
images originally generated by the GUI during the recording phase and updating 
coordinates for user input data; 

setting stronger conditions for the search for use in subsequent playbacks; and 
playing back the application program according to the second execution scenario 
with the stronger conditions for search. 

14. The method of claim 13, wherein the soft conditions comprise a first set of 
bounds for differences in shapes of contours, text and image content, or layout. 
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15. The method of claim 14, where the stronger conditions comprise a second 
set of bounds for differences in shapes of contours, text and image content, or layout, 
the second set being different than the first set. 

16. The method of claim 14, wherein GUI noise conditions comprise at least 
one of changeable color schemes, highlighting of items, noise from video sources, and 
anti-aliasing effects. 

17. The method of claim 14, wherein updating recorded images comprises using 
playback images as recorded images for subsequent playbacks. 

18. An article comprising: a machine accessible medium containing 
instructions, which when executed, result in automatically controlling execution of an 
application program having a GUI to reduce GUI noise by 

capturing user input data and images displayed by the GUI during a recording 
phase of execution of the application program; 

analyzing the captured user input data and recorded images to generate a first 
execution scenario during the recording phase; 

setting soft conditions for a search for the application program for the first 
execution scenario; 

generating simulated user input data based on the first execution scenario during 
a playback phase of execution of the application program and inputting the simulated 
user input data to the application program; 

performing image analysis on playback images displayed by the GUI as a result 
of processing the simulated user input data during the playback phase and the recorded 
images; 

updating the first execution scenario to form a second execution scenario to 
reduce GUI noise conditions observed during playback, including updating the recorded 
images originally generated by the GUI during the recording phase and updating 
coordinates for user input data; 

setting stronger conditions for the search for use in subsequent playbacks; and 
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playing back the application program according to the second execution scenario 
with the stronger conditions for search. 

19. The article of claim 18, wherein the soft conditions comprise a first set of 
5 bounds for differences in shapes of contours, text and image content, or layout. 

20. The article of claim 19, where the stronger conditions comprise a second set 
of bounds for differences in shapes of contours, text and image content, or layout, the 
second set being different than the first set. 
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21. The article of claim 18, wherein GUI noise conditions comprise at least one 
of changeable color schemes, highlighting of items, noise from video sources, and anti- 
aliasing effects. 
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22. The article of claim 1 8, wherein instructions to update recorded images 
comprise instructions to use playback images as recorded images for subsequent 
playbacks. 



